How do you batch insert/update large data sets efficiently?

Question

ICSM Computer · Answer

Efficiently batch inserting or updating large datasets in IndexedDB (especially 10,000+ records) requires careful use of transactions, chunking, and avoiding memory bottlenecks. Here's how you can do it
efficiently and safely:
1. Use Bulk APIs (bulkAdd,
bulkPut) with Dexie.js
If you're using Dexie.js, it provides bulkAdd and
bulkPut which are highly optimized.
await db.users.bulkAdd([
  { id: 1, name: "Alice" },
  { id: 2, name: "Bob" },
  // ...
]);
bulkAdd() → Inserts new items (fails on existing keys)bulkPut() → Inserts or updates (upsert)Automatically runs in a single transaction (very fast)
2. Chunk the Data (to avoid memory/transaction limits)
IndexedDB transactions have limits (browser- and platform-dependent). Safe chunk size:
500–1000 records per transaction.
async function chunkedInsert(data, chunkSize = 1000) {
  for (let i = 0; i andlt; data.length; i += chunkSize) {
    const chunk = data.slice(i, i + chunkSize);
    await db.users.bulkPut(chunk);
  }
}
⚠️ Without chunking, large sets (e.g. 100k records) may crash or freeze the browser.
3. Use Explicit Transactions (Vanilla IndexedDB)
If not using Dexie:
const tx = db.transaction(['users'], 'readwrite');
const store = tx.objectStore('users');
for (let user of users) {
  store.put(user); // or store.add(user)
}
tx.oncomplete = () =andgt; console.log("All inserted");
But this is slower and harder to manage than Dexie.
4. Update Efficiently Using modify() in Dexie
await db.users
  .where('status')
  .equals('inactive')
  .modify(user =andgt; {
    user.flagged = true;
  });
Efficient for bulk updating in-place using a predicate condition.
5. Avoid JSON.parse/stringify for Huge Sets
Handling 100K records as a full JSON array in memory is expensive. Process data in
streams or generators if source is large (e.g., from file or API).
6. Wrap in Transactions for Speed and Atomicity
Dexie:
await db.transaction('rw', db.users, async () =andgt; {
  await db.users.bulkPut(users);
});
Vanilla:
const tx = db.transaction(['users'], 'readwrite');
const store = tx.objectStore('users');
// loop over store.put(...)
Transactions batch writes and are faster than calling put outside a transaction.
7. Parallel vs Sequential Batching
Don’t write all chunks in parallel (can overload IndexedDB).
Prefer:
for (const chunk of chunks) {
  await db.users.bulkPut(chunk);
}
Avoid:
await Promise.all(chunks.map(chunk =andgt; db.users.bulkPut(chunk))); // ❌ risk of lock contention
Summary Table
Technique
Use When
Benefit
bulkPut() (Dexie)
Most common bulk insert/update
Fast, transactional
Chunking (1000 max)
10K+ records
Prevents crashes
modify()
Bulk update with filters
Fast, no full read
Explicit Transaction
Vanilla IndexedDB
Atomic, grouped writes
Sequential Chunks
Large data sets
Prevents lock contention

interview

How do you batch insert/update large data sets efficiently?

ICSM Computer

Can you answer this question?

1 Answers

1. Use Bulk APIs (`bulkAdd`, `bulkPut`) with Dexie.js

2. Chunk the Data (to avoid memory/transaction limits)

3. Use Explicit Transactions (Vanilla IndexedDB)

4. Update Efficiently Using `modify()` in Dexie

5. Avoid JSON.parse/stringify for Huge Sets

6. Wrap in Transactions for Speed and Atomicity

7. Parallel vs Sequential Batching

Summary Table

Liked By

Technique	Use When	Benefit
`bulkPut()` (Dexie)	Most common bulk insert/update	Fast, transactional
Chunking (1000 max)	10K+ records	Prevents crashes
`modify()`	Bulk update with filters	Fast, no full read
Explicit Transaction	Vanilla IndexedDB	Atomic, grouped writes
Sequential Chunks	Large data sets	Prevents lock contention

interview

How do you batch insert/update large data sets efficiently?

ICSM Computer

Can you answer this question?

1 Answers

1. Use Bulk APIs (bulkAdd, bulkPut) with Dexie.js

2. Chunk the Data (to avoid memory/transaction limits)

3. Use Explicit Transactions (Vanilla IndexedDB)

4. Update Efficiently Using modify() in Dexie

5. Avoid JSON.parse/stringify for Huge Sets

6. Wrap in Transactions for Speed and Atomicity

7. Parallel vs Sequential Batching

Summary Table

Liked By

1. Use Bulk APIs (`bulkAdd`, `bulkPut`) with Dexie.js

4. Update Efficiently Using `modify()` in Dexie